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ABSTRACT 



In spring 1995, Florida's Pensacola Junior College (PJC) 
undertook the third assessment of institutional effectiveness using an 
assessment plan developed between 1990 and 1993 . Outcomes were examined for 
the 1994-95 academic year with respect to the 51 goals, 147 objectives, and 
253 indicators of effectiveness identified by the assessment plan in 16 
functional areas. Area supervisors were requested to report on at least one 
objective and one indicator for each of the goals. Results of the process 
included the following: (1) 84 of the 253 indicators were selected to be 
reported for 1994-95, although with multiple reports for some indicators a 
total of 171 indicator reports were received; (2) 90.6% (n=155) of the 

indicators were shown to have been fully met in terms of designated levels of 
performance; and (3) indicators that were not met included achieving an 
annual 1% improvement in retention rates, maintaining college energy usage 
below state guidelines, and achieving a 70% satisfaction rate for financial 
aid services. As a result of the process, it was recommended that the 
assessment be continued in 1995-96; that the total number of indicator 
reports generated be increased; and that more product-oriented, as opposed to 
process-oriented, indicators be developed. Tables showing outcomes by 
indicator are included. (HAA) 
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Introduction: In an on-going effort to improve college operations and to improve the documentation 
of institutional effectiveness, a plan of assessment was developed inl990-1993. In the spring of 1 993 , 
an initial run of the Pensacola Junior College Institutional Effectiveness Plan was undertaken for at 
least a single indicator and objective for each goal in the plan. In the spring of 1994, the second 
institutional effectiveness assessment was made according to this plan. Following are the results of the 
third institutional effectiveness assessment undertaken in the spring of 1995. It must be noted that this 
report is not a final product, only the third step of an evolving process. 
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A Word of Caution: These indicators are intended to be data for affirmation of the mission of the 
college and for continued improvement of the institution. The success of the process requires a non- 
threatening environment in which the aggressive pursuit of evaluative data, both positive and negative, 
is encouraged. Consequently the achievement or non-achievement of a given indicator standard, while 
extremely important to the operational unit, is not as important to the purpose of institutional 
effectiveness assessment as is the fact that the college has examined that facet of the college’s 
productivity, has generated concrete information concerning it, and has proceeded to act on that 
information for the improvement of the college. 



EXECUTIVE SUMMARY 

Year 3 Institutional Effectiveness Assessment (1994 - 1995) 
(R-95-346, 6-1-95)) 



The third institutional effectiveness assessment was undertaken by the Office of Institutional Research 
and Effectiveness in the spring of 1995, for the 1994-1995 academic year. This used the plan 
developed by the Institution Effectiveness Taskforce between 199Q and 1993, which consisted in the 
third year evaluation of sixteen functional areas, 5 1 goals, 147 objectives and 253 possible indicators. 
For the third run, area supervisors were requested to select one objective and one indicator for each of 
the goals, although some selected more and others less. Reports of performance indicators for these 
objectives were received through May 1, 1995. A follow-up effort was made through May 15. A 
second follow-up was done through June 1 . The process was successfully competed this year for the 
third time. 

Findings: 

(1) A total of 171 indicator reports were received for 1994-1995 (some indicator standards were 
used by more than one unit.) 

(2) Of these, 155 or 90.6% were fully met in terms of designated levels of performance. 

(3) Sixteen did not meet the specified standards. 

(4) Data from these assessments were used to a high degree (99.4%). 

Conclusions: 

The central mission of the college is affirmed by the results of this assessment, unit-by-unit across 
the college. 

Recommendations: 

(1) The process must be continued in 1995-1996. 

(2) Continued simplification of the process is in order; especially in the reduction of the number 
of indicators. 

(3) Each institutional unit should strive to focus more of its institutional effectiveness efforts upon 
indicators that show basic college productivity respective of its mission, rather than process 
activity measures. Although processes of the college are important they represent means toward 
ends rather than true outcome productivity. 



ANALYSIS 



1 . It is given under the current plan that each functional area (n=16) can have multiple goals (n=5 1), 
which in turn may have multiple objectives (n=147) and multiple indicators (n=253). In addition 
there is the potential for multiple reports based on individual indicators. However in 1994-1995 
assessment, there was to have been an indicator reported for each goal; that would have been a 
minimum 51 indicator responses. There were in fact 84 individual indicators actually selected to 
be reported. Some received multiple responses for the same objective, some were single responses 
for a total of 171 indicator reports in all. Thus the indicator reporting rate was 33.3% for 1994-1995 
(that is the percentage of individual standards reported at least one time). A summary of these 
reports is presented in TABLE I. 

2. TABLE II gives detailed information on these data. Of the 171 total indicator standard reports 
received, 155 or 90.6% of the prescribed standards were met or exceeded. Of all the reports 
received, the reporters stated in 170 or 99.4% of these instances that the report data (whether the 
standard was achieved or not) was used by the reporting unit for its improvement 

3. TABLE El displays indicators that have yet to be examined either in 1992-1993, or 1993-1994, or 
1994-1995 but which must be assessed at least once during the five-year cycle of evaluation. 




3 



5 



CONCLUSIONS 



It is important to recognize that this evaluative process is intended to assess the assessment process 
itself as well as to assess college performance each year. 



1. Overall indicator response suggests that PJC u nits are meeting or exceeding prescribed 
performance levels for the objectives assessed: and as a result, are meeting college mission goals . 

2. This thir d run was adequate. The level of cooperation by the participating reporters was excellent 
in the third run. The reporting system and paper trail worked well and continued to have a minimal 
impact on college resources. 

3. Too many indicators were found to be process rather than product oriented. More product outcome 
measures are needed. 

4. An overreliance on satisfaction surveys was noted. There should be a curtailment of survey-based 
indicators and more emphasis placed on outcome measures of a quantitative nature. 

5. Subtle changes in some objectives and indicators continued to be made in the course of the 
process. This resulted in some indicator objectives having evolved into objectives more 
appropriate to unit operational plans or institution strategic plan objectives. Most of these were 
subjected to the formal review and approval process. 

6. Little opposition to the process trial non, second run, or the third run was noted, though lack of full 
understanding of the process and its purpose is frequently encountered. (Some difference in 
opinion seems to exist between those who see the process as a mirror of SACS Criteria and those 
who envision it as a test of college mission compliance.) Some see the process as an added burden. 
Others confuse it with Strategic Planning and the SACS reaffirmation processes. 



That: 



RECOMMENDATIONS 



1. The process be continued, simplified and refined. 

2. College leadership continue to emphasize the importance of the process. 

3. In 1995-1996, the total number of indicator reports must be increased, with special emphasis on 
those indicators that relate to outcome productivity measures rather than measures of internal 
processes (see TABLE Eli). 

4. Individual reporting units continue to utilize the results of these assessments for continuous 
improvement. 

5. Continued efforts must be made to educate PJC personnel on the institutional effectiveness 
process. 
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Investment rates not less than market. 


| Use accounts payable discounts. 


Inprove purchasing rpocedures. 


Review purchasing cent, for eff. 


Verify automated billing. 


Verify supplies efficiencies. 


Verify enrollment reports. A 


Verily enrollment reports. 


Non registration fees not less than other coL 


90%, 30 day collection of del. accts. 
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TABLE H. 



ANALYSIS OF REPORTED INSTITUTIONAL EFFECTIVENESS INDICATORS (a) 

1994-1995 
(YEAR 3.) 



TOTAL INDICATOR REPORTS RECEIVED 


1 


171 


100.0% 


PRODUCTIVITY INDICATORS REPORT RECEIVED 


9 


5.3% 


PRODUCTIVITY INDICATORS REPORTS ACHIEVED 


|77.8% 


7 


4.1% 


PROCESS INDICATOR REPORTS RECEIVED 


162 * 


94.2% 


PROCESS INDICATORS ACHIEVED 


|95.5% 


148 


86.5% 


AT T. INDICATORS ACHIEVED 


155 


86.5% 


INDIVIDUAL INDICATORS DATA USED (of a possible 252) 


84 


33.3% 



(a) Source: Tabulated from TABLE I. data. 
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TABLE m 

LIST OF UNREPORTED INDICATORS THROUGH 1994-1995* 



New I. A. l.b. 


m.B.2.d. 


VI.B.3.a. 


Xm.C.3.a. 


XVI.F.l.c 


NewI.A.l.c. 


IV.A.2.a. 


Vn.A.l.a. 


xm.c.3.b. 


XVI.F.2.a. 


New I.A.l.d. 


IV.A.3.b. 


Vn.B.l.a. 


Xm.C.4.a. 


XVI. G. l.a. 


New I.A_2.a. 


IV.A.3.C. 


VHB.l.b. 


xm.c.4.b. 


XVLHJ.a 


New I.A_3.a. 


IV.A.3.d. 


DCA.3.a. 


XTV.C.2.a. 




New I.A_3.b. 


IV.A.3.e. 


DC A. 5, a. 


XDCC.2.a. 




New I.E.2.a. 


IV.B.l.a. 


XI.B.l.a. 


XK.C.3.b. 




New XVI.H.2.b. 


IV.B.3.b. 


XI.C.3.a. 


XIV.C.4.a. 




New XVI.B.2.b. 


IV.B.3.C. 


XHA.2.C. 


XV.A_3.a. 




I.B.l.a. 


IV.B.3.d. 


Xn.B.2.a. 


XV.A.5.b. 




I.C.l.b. 


IV.B.3.e. 


xm.A.i.b. 


XVI.C.l.a. 




I.C.2.a. 


IV.C.3.a. 


M.A.I.C. 


XVI.C.l.b. 




LD.l.b. 


IV.C.4.a. 


Xm.A.3.a. 


XVI.C.2.a. 




HA. La. 


IV.C.4.b. 


Xm.A.3.b. 


XVI.C.4.a. 




m.A.l.b. 


IV.D.l.b. 


Xm.A.4.a. 


XVI.C.4.C. 




m.A.i.c. 


IV.D.2.a. 


Xm.A.4.b. 


XVI.C.5.a. 




m.A.l.d. 


IV.D.3.a. 


XHI.A.4.C. 


XVI.C.5.b. 




m.A.l.e. 


IV.D.3.b. 


Xm.A.5.a. 


XVI.C.6.a. 




m.A.2.a. 


V.A.l.a. 


Xm.A.5.b. 


XVI.D.l.a. 




m.A.2.b. 


V.A.3.a. 


Xm.A.6.b. 


XVI.D.l.b. 




m.A.2.b. 


V.B.l.a.(l) 


xm.B.i.c. 


XVLD.3.a. 




m.A.2.c. 


V.B.l.a.(3) 


Xin.B.2.a. 


XVI.E.l.b. 




m.A.2.d. 


V.C.l.c. 


XHI.B.2.C. 


XVI.E.2.a. 




m.B.i.c. 


V.C.l.d. 


Xffl.C.l.c. 


XVI.E.2.b. 




m.B.l.d. 


V.C.l.e. 


XIII.C.2.a. 


XVI.E.3.a. 




m.B.2.a. 


V.C.2.a. 


Xin.C.2.b. 


XVI.E.3.b. 




m.B.2.b. 


V.C.4.a. 


Xffl.C.2.c. 


XVI.E.3.C. 




m.B.2.c 


VI.A.5.a. 


xm.c.2.d. 


XVI.F.l.b. 





* As of 6-1-95 
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